feat(openinfer): add OpenInfer as an OpenAI-compatible provider by AJ-ing · Pull Request #41034 · BerriAI/litellm

AJ-ing · 2026-09-14T03:35:09Z

TLDR

Problem this solves:

OpenInfer is not a native LiteLLM provider
Users have to fake it as a generic OpenAI-compatible endpoint
Spend tracking has no OpenInfer catalog rates

How it solves it:

Registers openinfer as a JSON OpenAI-compatible provider
Routes openinfer/<model> to https://api.openinfer.ai/v1
Stores vendor $/1M rates as per-token catalog prices

User Flow

Before: a developer who wants OpenInfer through LiteLLM has no native provider, so the call never reaches https://api.openinfer.ai/v1

They set OPENINFER_API_KEY and send POST https://litellm-domain/v1/chat/completions with "model": "openinfer/@oi/Llama-3.2-1B-Instruct"
LiteLLM does not resolve openinfer as a provider
The request never hits https://api.openinfer.ai/v1/chat/completions

After: the same request routes to OpenInfer's chat completions endpoint with the catalog model id and list price

They set OPENINFER_API_KEY and send POST https://litellm-domain/v1/chat/completions with "model": "openinfer/@oi/Llama-3.2-1B-Instruct"
LiteLLM strips the openinfer/ prefix and uses https://api.openinfer.ai/v1
The upstream call is POST https://api.openinfer.ai/v1/chat/completions with "model": "@oi/Llama-3.2-1B-Instruct"

Relevant issues

Closes #34357
Replaces #34623, which GitHub closed when litellm_internal_staging was deleted and recreated. The provider code never landed on main or staging

Companion docs: BerriAI/litellm-docs#905

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/llms/openai_like/test_openinfer_provider.py -v (12 passed; catalog test no longer pins vendor rates)
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

No OPENINFER_API_KEY in this environment, so this is routing proof only. Live chat completions against the vendor still need a key

Before (`ed4caeb`)

python3 -c 'import json; print("openinfer" in json.load(open("litellm/llms/openai_like/providers.json")))'
Observed: False. main also has no LlmProviders.OPENINFER enum

After (rebased tip)

python3 - <<'PY' from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider model, provider, api_key, api_base = get_llm_provider( model="openinfer/@oi/Llama-3.2-1B-Instruct", custom_llm_provider=None, api_base=None, api_key="sk-test", ) print(f"model={model}") print(f"provider={provider}") print(f"api_base={api_base}") PY
Observed:

model=@oi/Llama-3.2-1B-Instruct
provider=openinfer
api_base=https://api.openinfer.ai/v1

Type

New Feature

Caveats (if any)

Medium

Docs CI may stay red until docs(openinfer): add OpenInfer provider page litellm-docs#905 lands on litellm-docs main
- LiteLLM's documentation job reads env-var names from that repo
Live /v1/chat/completions proof is not in this description
- This checkout has no OPENINFER_API_KEY

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

codspeed · 2026-09-14T03:38:05Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing AJ-ing:feat-openinfer-provider (2b67e89) with main (6f5ca84)}

greptile-apps · 2026-09-14T03:38:13Z

Confidence Score: 5/5

[Medium risk] Adds OpenInfer as a new LLM provider.

The PR appears safe to merge; no outstanding findings remain.

Summary

The PR registers OpenInfer as an OpenAI-compatible chat provider and adds model pricing and capability data.

Adds the provider route, credentials, endpoint declarations, and proxy creation fields.
Adds tests for routing, request configuration, and catalog consistency.

_{Reviews (6) · Last reviewed commit: "fix(openinfer): assert catalog invariant..."}

codecov · 2026-09-14T03:49:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

AJ-ing · 2026-09-15T01:27:05Z

Rebased onto latest litellm_internal_staging and dropped the constants.py comment Greptile flagged. Also pinned the outgoing chat request contract in the provider tests. @greptileai

onkarbpatil · 2026-09-15T16:53:37Z

@AJ-ing what else is left to get this reviewed and merged?

AJ-ing · 2026-09-16T01:06:09Z

@onkarbpatil Just rebased onto latest litellm_internal_staging again (ahead 7 / behind 0).

What's left for merge is maintainer review. I don't have merge access.

Status:

Greptile 5/5, Veria clean, provider tests green
Companion docs still open: docs(openinfer): add OpenInfer provider page litellm-docs#905 (helps the documentation/env-var scan stay green)
The only red check on this PR is Vertex AI / Run tests, and that same job is also red on staging itself (gemini embeddings cost asserts). Not from the OpenInfer changes

@mateo-berri when you have a minute, this one and the docs PR are ready for a look

onkarbpatil · 2026-09-18T19:19:30Z

@AJ-ing @mateo-berri any update on review?

AJ-ing · 2026-09-19T00:56:18Z

@onkarbpatil No new code changes. Branch is current with staging, Greptile 5/5. Still needs @mateo-berri review; docs companion is BerriAI/litellm-docs#905.

onkarbpatil · 2026-09-21T19:45:11Z

@mateo-berri ?

chongyihu · 2026-09-28T22:47:30Z

@mateo-berri @ryan-crabbe-berri @yuneng-berri @kerry-berri
Sorry for the repeated pings! Could you help us get this reviewed and approved? Let me know if anything's missing or if there's anything I can do to help move it along.

AJ-ing · 2026-10-01T00:07:06Z

@onkarbpatil @chongyihu Rebased onto latest main after staging was deleted (conflicts cleared; PR is MERGEABLE again). Tip is 64be726. Provider unit tests: 12 passed. Waiting on maintainer review / CI. @greptileai

AJ-ing · 2026-10-01T01:34:57Z

Pushed 7d016c9: catalog unit test now checks backup agreement and positive rates instead of pinning vendor $/token (Greptile P2).

Remaining red checks look like main noise, not OpenInfer: CLICKHOUSE_FLUSH_INTERVAL_SECONDS docs scan, osv lockfile CVEs, interactions OpenAPI tests, batches metadata asserts, proxy-behavior postgres role. Re-requesting review. @greptileai

chongyihu · 2026-10-01T19:09:19Z

Hi @AJ-ing , seems like there are still upstream CI issues. Is the upstream resolved? If it is resolved, can you hep us rebase and rerun the CI test? If it is not, is it possible that you guys can manually approve and merge the openinfer integration? It would be very helpful!

AJ-ing · 2026-10-02T05:34:36Z

@chongyihu Rebased onto latest main (tip ed7917d). OpenInfer unit tests still 12/12. CI should re-run now; mergeability should clear once checks finish.

AJ-ing · 2026-10-02T05:35:05Z

@greptileai Rebased onto latest main. Please re-review the tip.

Register OpenInfer in the JSON provider registry with catalog models, proxy create fields, and unit coverage so requests route via openinfer/ Co-authored-by: Cursor <cursoragent@cursor.com>

Pinning per-token prices made CI fail when the vendor list changed. Keep the backup agreement and positivity checks our code owns. Co-authored-by: Cursor <cursoragent@cursor.com>

AJ-ing · 2026-10-06T01:32:49Z

@kerry-berri Rebased onto latest main (tip 2b67e89). OpenInfer unit tests still 12/12. Ready for your approval.

AJ-ing · 2026-10-06T01:32:51Z

@greptileai Tip rebased onto latest main. Please re-review.

AJ-ing · 2026-10-06T01:48:44Z

@kerry-berri Both remaining reds are also failing on current main (not OpenInfer):

misc: test_seed_tracing_fixtures fixture hash / bulk-export (same on main run 37399367833)
proxy-infra-root: test_component_allowlists missing /v1/logs allowlist (same on main)

OpenInfer unit tests are green (12/12). Diff only touches provider registry / catalog / our test file.

AJ-ing requested a review from mateo-berri as a code owner September 14, 2026 03:35

AJ-ing mentioned this pull request Sep 14, 2026

feat(openinfer): add OpenInfer as an OpenAI-compatible provider #34623

Closed

1 of 3 tasks

greptile-apps Bot reviewed Sep 14, 2026

View reviewed changes

Comment thread litellm/constants.py Outdated

Comment thread tests/test_litellm/llms/openai_like/test_openinfer_provider.py

devin-ai-integration Bot mentioned this pull request Sep 14, 2026

feat(pricing): add azure gpt-chat-latest rates and drop retired friendliai llama-3.1 entries #40976

Merged

4 of 6 tasks

AJ-ing force-pushed the feat-openinfer-provider branch from 80299be to b6414df Compare September 15, 2026 01:26

AJ-ing force-pushed the feat-openinfer-provider branch from b6414df to 907d359 Compare September 16, 2026 01:06

devin-ai-integration Bot changed the base branch from litellm_internal_staging to main September 23, 2026 14:34

devin-ai-integration Bot requested review from kerry-berri, ryan-crabbe-berri and yuneng-berri as code owners September 23, 2026 14:34

devin-ai-integration Bot requested a review from a team September 23, 2026 14:34

BerriAI deleted a comment from devin-ai-integration Bot Sep 30, 2026

AJ-ing force-pushed the feat-openinfer-provider branch from 907d359 to 64be726 Compare October 1, 2026 00:05

greptile-apps Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread tests/unit/llms/openai_like/test_openinfer_provider.py Outdated

AJ-ing force-pushed the feat-openinfer-provider branch from 7d016c9 to ed7917d Compare October 2, 2026 05:34

AJ-ing and others added 2 commits October 6, 2026 12:31

feat(openinfer): add OpenInfer as OpenAI-compatible provider

094dba0

Register OpenInfer in the JSON provider registry with catalog models, proxy create fields, and unit coverage so requests route via openinfer/ Co-authored-by: Cursor <cursoragent@cursor.com>

fix(openinfer): assert catalog invariants instead of vendor rates

2b67e89

Pinning per-token prices made CI fail when the vendor list changed. Keep the backup agreement and positivity checks our code owns. Co-authored-by: Cursor <cursoragent@cursor.com>

AJ-ing force-pushed the feat-openinfer-provider branch from ed7917d to 2b67e89 Compare October 6, 2026 01:32

Conversation

AJ-ing commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Pre-Submission checklist

Screenshots / Proof of Fix

Before (ed4caeb)

After (rebased tip)

Type

Caveats (if any)

Medium

Final Attestation

Uh oh!

codspeed Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

greptile-apps Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 14, 2026

Codecov Report

Uh oh!

AJ-ing commented Sep 15, 2026

Uh oh!

onkarbpatil commented Sep 15, 2026

Uh oh!

AJ-ing commented Sep 16, 2026

Uh oh!

onkarbpatil commented Sep 18, 2026

Uh oh!

AJ-ing commented Sep 19, 2026

Uh oh!

onkarbpatil commented Sep 21, 2026

Uh oh!

chongyihu commented Sep 28, 2026

Uh oh!

AJ-ing commented Oct 1, 2026

Uh oh!

Uh oh!

AJ-ing commented Oct 1, 2026

Uh oh!

chongyihu commented Oct 1, 2026

Uh oh!

AJ-ing commented Oct 2, 2026

Uh oh!

AJ-ing commented Oct 2, 2026

Uh oh!

AJ-ing commented Oct 6, 2026

Uh oh!

AJ-ing commented Oct 6, 2026

Uh oh!

AJ-ing commented Oct 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AJ-ing commented Sep 14, 2026 •

edited

Loading

Before (`ed4caeb`)

codspeed Bot commented Sep 14, 2026 •

edited

Loading

greptile-apps Bot commented Sep 14, 2026 •

edited

Loading